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Abstract 

With the recent observation of graphene-like Landau levels at the surface of topologi- 
cal insulators, the possibility of fractional quantum Hall effect, which is a fundamental 
signature of strong correlations, has become of interest. Some experiments have re- 
ported intra-Landau level structure that is suggestive of fractional quantum Hall effect. 
This paper discusses the feasibility of fractional quantum Hall effect from a theoretical 
perspective, and argues that while this effect should occur, ideally, in the n = and 
|n| = 1 Landau levels, it is ruled out in higher |n| Landau levels. Unlike graphene, 
the fractional quantum Hall effect in topological insulators is predicted to show an 
interesting asymmetry between n = 1 and n = — 1 Landau levels due to spin-orbit 
coupling. 
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1. Introduction 

The fractional quantum Hall effect (FQHE) is a fundamental signature of strong 
correlations in two dimensional systems. In the FQHE, the electrons lower their in- 
teraction by binding with vortices to form composite fermions 01 01 which are then 
weakly interacting. The integer quantum Hall effect of these composite fermions is 
the FQHE of the electrons. The formation of composite fermions and the FQHE 
relies on an ineraction between electrons that is sufficiently short range. The com- 
posite fermion wavefunction is a good approximation when this interaction is suffi- 
ciently short ranged, a condition which can be quantified by the Haldane pseudopoten- 
tials. QiiS 

With rapid improvement in the quality of topological insulators (TIs) has come 
progress towards the observation of integer and fractional quantum Hall effects in these 
systems. Topological insulators have insulating bulk electronic states but an odd num- 
ber of topologically protected Dirac cones on the surface. [6, 7, 8] Because of the Dirac 
nature of the electrons, many properties of TIs are analagous to those of graphene. One 
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of these properties is the Landau level (LL) structure for Dirac fermions, which is quite 
unique; the energy levels are proportional to the square root of the LL indices due to 
the chiral nature of the electrons. H ficj [III [T2I [Till Recently this graphene-like integer 
quantum Hall effect has been observed in strained bulk HgTe lfl4l1 and the LL spectrum 
has been measured in Bi2Se3 using scanning tunneling microscopy [15, T^. In ad- 



dition, features in the Hall resistance at fractional filling factors have been speculated 
to be related to the FQHE of topological insulators. 1171,118 1 With these experiments, 
coupled with the observation of the FQHE in graphene [19, 20] and the evidence for 
a strong inter-electron interaction in TIs 112111 . arises the question of the possiblity of 
observing the FQHE in topological insulators. 

This paper reports on the stability of the FQHE in topological insulators from a 
theoretical perspective. It is shown that the FQHE is observable for n = and \n\ = 1 
LLs, however is not stable for higher LLs. Due to the spin orbit interaction of the TIs, 
there is an anisotropy of the \n\ = 1 LLs which is not seen in graphene or in the lowest 
(n = 0) Landau level. 112.21 \23l l24l 12511 These results hold when the bulk is conducting, 
although a bulk contribution to the conductivity may make these features more difficult 
to observe. 



2. Theory 



The Landau levels of topological insulators have been calculated [27, 26] and bear 



a striking similarity to those of graphene I22I 0, [l2 , 23 ] . There are however two im 



portant differences. First, topological insulators have a high Zeeman energy, with a 
g-factor as large as 30 (Ref. 112811 ). Second, although the Hamiltonians of a TI and 
graphene are mathematically identical, the Hilbert space basis functions are different. 
In graphene, spin up and spin down states are degenerate, and the Hamiltonian is writ- 
ten in the pseudospin basis, which is related to the presence of two atoms in the unit 
cell of the hexagonal lattice. In TIs, the Hamiltonian is written in the basis of the real 
spin. These two factors manifest as a tendency for the spin to polarize along the direc- 
tion of magnetic field in TIs. This work considers the ideal case of a single Dirac cone, 
occuring in materials such as Bi2Se3 and Bi2Te3, 1I29II30I1 and restrict the calculation 
to a single surface. If the two surfaces are not too close together, the final results will 
hold for each surface independently. 

The stability of the FQHE states can be predicted by considering the ratio of the 
Haldane pseudopotentials, which are the matrix elements of the interaction of the elec- 
trons within a LL, 12, |3l U, |5[] 

v c t = (* nm |y|*„o) (i) 

where \^ n m) is the state in the nth LL with angular momentum — Km and V is the 
interaction. These pseudopotentials measure the interaction between particles in two 
states within the same LL. The two-body states must be antisymmetric which implies 
that only pseudopotentials of odd relative angular momentum, (—hm where m is odd) 
are relevant. Typically one can predict the stability of the FQHE by considering only 
the m — 1 and m — 3 pseudopotentials, since falls off quickly as m increases. 
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Although there are ways to more precisely quantify the pseudopotentials' role in pre- 
dicting the stability of the composite fermions, these states tend to be stable if the ratio 
of the m = 1 pseudopotential to the m = 3 pseudopotential is larger than 1.3 — 1.5 
(Ref. ||2|,|3l|4j,|5|]); in this case the FQHE may be observed. This is because the com- 
posite fermion wavefunction is accurate when the interaction is short range (V^ ^ 0, 
V*g = 0), and remains a good approximation when Vjg/V3{ is large enough. 

The non-interacting electron Hamiltonian in the presence of a perpendicular mag- 
netic field is [26] 

H = vpcr ■ II + gfiBBa z (2) 

where vf is the Fermi velocity and II = p + eA/c is the two-dimensional canonical 
momentum. The z-direction is taken to be perpendicular to the surface and the sym- 
metric gauge, A = (B/2)(—y, x, 0) T , has been used. Here, the superscript T stands 



for transpose. In analogy with graphene 1221 11211 this Hamiltonian can be written in 
terms of ladder operators, 

H — ( 9^bB -ihvF^VZ a \ 
\ ihvpl B ~ V2aJ —g/isB J 

where £b = \J eB/Hc is the magnetic length and 



a y/2\2t B 



(4) 



are the ladder operators. Here, z = x + iy, as usual. There are two more linearly 
independent ladder operators, yfl 
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(5) 



which are related to the z-component of the angular momentum, L z — —h(b^b — 
a) a). In analogy with graphene, the solutions are linear combinations of the usual two 
dimensional electron gas (2DEG) eigenstates, yfl 4>nm, which satisfy a^a<j) nm — ncf> nm 
and tfb(f) nm = m^ nm , 



where 



— zsgn(n) cos <p n 
^/2(l-sgn(n) sin ip n ) 

n = 



\/(l - sgn(n) siny;„)/2 n^O 
1 n = 



n ^ 

(7) 
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tp n = gHBB/{hvpl^'^ s /2 \n\) and sgn(n) = nj \n\. These states have energies ll26ll 

(8) 



E n =sgn(n)y , (.g MB B) 2 + 2\n\h 2 v 2 F £ B 2 n + 
Eo = - (j^bB 



In the limit of g — > 0, the above equations reduce to those of graphene. 
The Coulomb interaction is 



V(]r 1 -r 2 \) = - r ^ , (9) 

e ri - r 2 



which acts on the two-body states, 

\\nm 1 \nm 2 )) = $„ mi 8* ra2 

= a 2 n \\n\ - lmi t; |n| - 1 m 2 t) + 

+ a„^„||n| mi |; |n| - 1to 2 t> + (10) 
+ a„/3„||n| - lmi t; M to 2 i) + 
+ /3n||n| mi |; |n| m 2 |) 

where the single ket denotes the product state of the usual quantum well states, |ni mi ; n 2 m 2 ) 
4'n 1 m 1 ® 4>n 2 m 2 - Then the pseudopotentials of the Coulomb interaction are 



V^' m = ((nrni;nm 2 ||F||nm3;nm 4 )) 



4(1 - sgn(n)sm<^„) 2 m 
+ ~(1 - sgn(n) sin^) 2 ^ + \ ^ 2 Vn Vt' n ~ X) 



(11) 



where JHH 



= (|n| mi; |n| m 2 |l / ||n| m.3; |nj 7714) 



d 2 fc 



/ I fc2 



n 2 



L m {k 2 )e~ k (12) 



(2tt) 2 

= (| n | mi;|n|-lm 2 |V||n| m 3 ;|n|-lm 4 ) 

= / ^V(fc)L w (y) L W _! (y) i m (fc 2 )e" fc2 (13) 

Here, V(fc) is the Fourier transform of the Coulomb potential and — Turn is the relative 
angular momentum. 



3. Results and discussion 

Figure Q] shows the ratio of the m = 1 to the m = 3 pseudopotentials for the 
Coulomb interaction as a function of Zeeman energy. In the lowest (n = 0) LL, the 
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Figure 1: The ratio of the first and third relative angular momentum pseudopotentials for the Coulomb 
interaction in the lowest 7 LLs as a function of Zeeman energy. The solid lines are for positive n while the 
dashed lines are for negative n. The inset is the same as the main figure, but with a different vertical scale to 
highlight the \n\ = 2, 3 LLs. 
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Figure 2: The ratio of the first and third relative angular momentum pseudopotentials as a function of Zeeman 
energy for an interaction which models the effect of screening due to the bulk electrons (see text). The solid 
lines are for positive n while the dashed lines are for negative n. The inset is the same as the main figure, 
but with a different vertical scale to highlight the |n| = 2, 3 LLs. 



spin is perfectly polarized without the addition of a magnetic field, so the ratio is in- 
dependent of the Zeeman energy. For all other values of |n|, the symmetry between n 
and —n is broken as the Zeeman energy increases from zero. The pseudopotential ratio 
increases as a function of Zeeman energy for n > 0, indicating an increased stability of 
the fractional quantum Hall state, while just the opposite happens for n < 0. However, 
when n > 2, the pseudopotential ratio becomes smaller than 1 .2, and the fractional 



Hall effect is likely to be unstable. Thus the features seen in Refs. 1171 11811 at filling 
factors n > 2 are unlikely to be precursors to the FQHE. 

The presence of a noninsulating bulk may significantly alter the form of the inter- 
action. As a model, consider the form 

V(r) = C ] (14) 

ery/1 + (r/r d ) 4 

which behaves as the Coulomb interaction (V ~ 1/r) at short distances and as a dipole - 
dipole interaction (V ~ 1/r 3 ) at large distances. The parameter characterizes the 
size of the dipole. For simplicity, I have taken r<i = 1/ &tf; when the two electrons are 
much further than a Thomas-Fermi screening length, then they interact as if they are 
two dipoles, with the noninsulating bulk creating the positive end of each dipole. 
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The pseudopotential will have exactly the same form as Equations ([TTli-([T3l with 
V(k) replaced by the Fourier transform of the model interaction, (fl4b . Figure[2]shows 
the ratio of the m = 1 to the m = 3 pseudopotential for the model interaction. Fig- 
ure sQ] and |2] are qualitatively similar, and differ only in quantitative aspects. The ratio, 
VTa /VCff > has increased for the n = 0, 1, and — 1 LLs, which suggests an increased 
stability of the FQHE. However, the ratio has decreased for n > 2; i.e. the noninsulat- 
ing bulk will make the FQHE even less stable. 

4. Conclusion 

In conclusion, this paper has ruled out FQHE in LLs \n\ > 2, while predicting its 
stability for n = 0, 1, —1 for a TI surface. Due to the strong spin orbit interaction in 
TIs, there is an asymmetry in the FQHE for n = 1 and n = — 1 LLs, in contrast to 
graphene. The effect of screening by a noninsulating bulk has been included by using 
a model interaction which looks like a dipole-dipole interaction at distances larger than 
the Thomas Fermi screening length. Although the composite fermion states in the LLs 
n = 0, 1, —1 are further stabilized by this model interaction, the composite fermion 
state in higher LLs (|n| > 2) are further destabilized due to the weaker long range 
interaction. 
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